Mình đã viết nhiều bài về việc tạo PVC trên eks hay k8s
1) Install CSI Driver SMB
Vậy nếu trong cluster k8s của bạn có cả linux và windows node thì phải làm sao?
vâng chúng đó có 1 giải pháp đó là CSI Driver SMB
https://github.com/kubernetes-csi/csi-driver-smb
https://www.phillipsj.net/posts/how-to-use-the-windows-csi-proxy-and-csi-smb-driver-for-kubernetes/
Đầu tiên bạn cần cài đặt CSI Driver SMB thông qua helm chart.
Mình cung cấp 1 file application của argocd
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: csi-driver-smb-nimtechnology-staging
namespace: argocd
spec:
destination:
namespace: kube-system
name: 'arn:aws:eks:us-west-2:04370111XXXXX:cluster/dev-mdcl-nimtechnology-engines'
project: meta-structure
source:
repoURL: https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
targetRevision: "v1.9.0"
chart: csi-driver-smb
helm:
values: |-
windows:
dsName: csi-smb-node-win
enabled: true
Check the windows node on the os for the existence of the following directory once the deployment has finished:
C:\var\lib\kubelet\plugins\smb.csi.k8s.io
2) Install SMB Server.
Tiếp theo bạn sẽ cần cài đặt SMB Server.
Trong bài test này chúng ta sẽ cài smb server trong cluster k8s
https://github.com/kubernetes-csi/csi-driver-smb/tree/master/deploy/example/smb-provisioner
The first step is to add the secret:
kubectl create secret generic smbcreds --from-literal username=windows --from-literal password="IsAwesome"Tiếp đến là bạn cài đặt smb-server
Cách 1: dụng local disk of node
modify /smbshare-volume in deployment to specify different path for smb share data store
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/example/smb-provisioner/smb-server.yaml---
kind: Service
apiVersion: v1
metadata:
name: smb-server
labels:
app: smb-server
spec:
type: ClusterIP # use "LoadBalancer" to get a public ip
selector:
app: smb-server
ports:
- port: 445
name: smb-server
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: smb-server
spec:
replicas: 1
selector:
matchLabels:
app: smb-server
template:
metadata:
name: smb-server
labels:
app: smb-server
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: smb-server
image: andyzhangx/samba:win-fix
env:
- name: PERMISSIONS
value: "0777"
- name: USERNAME
valueFrom:
secretKeyRef:
name: smbcreds
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: smbcreds
key: password
args: ["-u", "$(USERNAME);$(PASSWORD)", "-s", "share;/smbshare/;yes;no;no;all;none", "-p"]
volumeMounts:
- mountPath: /smbshare
name: data-volume
ports:
- containerPort: 445
volumes:
- name: data-volume
hostPath:
path: /home/kubernetes/smbshare-volume # modify this to specify another path to store smb share data
type: DirectoryOrCreate
Cách 2: Create a Samba Server deployment on the network disk
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/example/smb-provisioner/smb-server-networkdisk.yaml---
kind: Service
apiVersion: v1
metadata:
name: smb-server
labels:
app: smb-server
spec:
type: ClusterIP # use "LoadBalancer" to get a public ip
selector:
app: smb-server
ports:
- port: 445
name: smb-server
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-networkdisk-smbshare
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
#storageClassName: default # storage provider is Azure disk in Azure and persistent disk in GCE.
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: smb-server
spec:
replicas: 1
selector:
matchLabels:
app: smb-server
template:
metadata:
name: smb-server
labels:
app: smb-server
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: smb-server
image: dperson/samba
env:
- name: PERMISSIONS
value: "0777"
- name: USERNAME
valueFrom:
secretKeyRef:
name: smbcreds
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: smbcreds
key: password
args: ["-u", "$(USERNAME);$(PASSWORD)", "-s", "share;/smbshare/;yes;no;no;all;none", "-p"]
volumeMounts:
- mountPath: /smbshare
name: data-volume
ports:
- containerPort: 445
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: pvc-networkdisk-smbshare
Ok giờ chúng ta test pvc với smb.
Đầu tiêu bạn cần tạo 1 storageclass và có cách thông tin connect smb server.
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: smb provisioner: smb.csi.k8s.io parameters: # On Windows, "*.default.svc.cluster.local" could not be recognized by csi-proxy source: "//smb-server.default.svc.cluster.local/share" # if csi.storage.k8s.io/provisioner-secret is provided, will create a sub directory # with PV name under source csi.storage.k8s.io/provisioner-secret-name: "smbcreds" csi.storage.k8s.io/provisioner-secret-namespace: "<namespace>" # you need to change it csi.storage.k8s.io/node-stage-secret-name: "smbcreds" csi.storage.k8s.io/node-stage-secret-namespace: "<namespace>" # you need to change it volumeBindingMode: Immediate mountOptions: - dir_mode=0777 - file_mode=0777 - uid=1001 - gid=1001
3) Demo
Giờ bạn tạo 1 pvc và deployment
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-smb
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: smb
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pwsh-smb
labels:
app: pwsh
spec:
replicas: 1
template:
metadata:
name: pwsh
labels:
app: pwsh
spec:
nodeSelector:
"kubernetes.io/os": windows
containers:
- name: pwsh
image: mcr.microsoft.com/powershell:lts-nanoserver-1809
command:
- "pwsh.exe"
- "-Command"
- "while (1) { Add-Content -Encoding Ascii C:\\mnt\\smb\\data.txt $(Get-Date -Format u); sleep 1 }"
volumeMounts:
- name: smb
mountPath: "/mnt/smb"
subPath: subPath
volumes:
- name: smb
persistentVolumeClaim:
claimName: pvc-smb
selector:
matchLabels:
app: pwsh
Then you can test by verifing that data.txt exists in the SMB share.
$ kubectl exec -it pwsh-smb-0 -- pwsh
C:/ $ ls mnt/smb
Directory: C:\mnt\smb
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/11/2022 8:27 PM 3476 data.txt
Pay attention to
Bạn sẽ thấy là khi bạn tạo storageclass mà sử dụng source: "//smb-server.default.svc.cluster.local/share"
thì bạn sẽ thấy CSI Driver SMB sẽ không resolve được smb-server.default.svc.cluster.local nên bạn chỉ có thể điền thằng IP hoặc domain public vào storageclass
https://github.com/kubernetes-csi/csi-driver-smb/issues/294
If you take care about Bandwidth and IOPS and not take care about recrycle files that were deteled.
Nếu bạn để ý.
Khi mà các file được xóa trong PVC thì sẽ được SMB backup trong folder tên là .delete
Việc recycle/backup file đã được delete thì nó khá tốn bandwidth and IOPS của Volume
Với cài IOPS này thì nó đang vượt ra cả max IOPS của GP3
Và mình sẽ cần phải disable recycle/backup file đã được deleled in SMB server
https://github.com/mrnim94/samba/blob/master/Dockerfile#L36-L40
và bạn có thể tham khảo image sau
mrnim94/samba:0.0.2-not-recycle-0.0.3
Windows clients will not refresh Linux samba file locally
Như bạn cũng đã thấy là mình sử dụng smb-server để cung cấp PVC(readwritemany) for windows pods.
Nó gặp 1 vấn là khi linux client save 1 file “nim.txt” xuống SMB server và đã xong thì Client windows vẫn chưa thấy file đó trên smb-server
Mình đã tìm ra link này:
https://unix.stackexchange.com/questions/425934/windows-clients-will-not-refresh-linux-samba-file-locally-if-reading-file-at-int/426224#426224
I resolved this by placing
oplocks = Falsein my smb.conf under my share settings.
https://www.samba.org/samba/docs/old/Samba3-HOWTO/locking.html#id2615926
Bạn có thể tìm hiểu Oplocks
https://www.oreilly.com/openbook/samba/book/ch05_05.html
Opportunistic Locking
Opportunistic locking allows a client to notify the Samba server that it will not only be the exclusive writer of a file, but will also cache its changes to that file on its own machine (and not on the Samba server) in order to speed up file access for that client. When Samba knows that a file has been opportunistically locked by a client, it marks its version as having an opportunistic lock and waits for the client to complete work on the file, at which point it expects the client to send the final changes back to the Samba server for synchronization.
If a second client requests access to that file before the first client has finished working on it, Samba can send an oplock break request to the first client. This tells the client to stop caching its changes and return the current state of the file to the server so that the interrupting client can use it as it sees fit. An opportunistic lock, however, is not a replacement for a standard deny-mode lock. It is not unheard of for the interrupting process to be granted an oplock break only to discover that the original process also has a deny-mode lock on a file as well. Figure 5.8 illustrates this opportunistic locking process.
Figure 5.8: Opportunistic locking
In terms of locks, we highly recommend using the defaults provided by Samba: standard DOS/Windows deny-mode locks for compatibility and oplocks for the extra performance that local caching allows. If your operating system can take advantage of oplocks, it should provide significant performance improvements. Unless you have a specific reason for changing any of these options, it’s best to leave them as they are.
ChatGPT answered:
In a Windows client-Linux Samba server environment, it’s possible that Windows clients are not refreshing files that are stored on the Samba server. This issue is usually caused by the cache on the Windows client machine.
To resolve this issue, you can try the following steps:
- Disable Opportunistic Locking: Opportunistic Locking is a feature in Windows that caches file and folder data on a client computer to improve performance. Disabling this feature can prevent the cache from causing the file not to refresh. You can disable this feature by setting the “oplocks” parameter in the smb.conf file on the Samba server to “False”.
- Clear the Windows cache: Clearing the Windows cache can also resolve the issue. To do this, you can use the “ipconfig /flushdns” command in the Command Prompt.
- Restart the SMB service: Restarting the SMB service on the Windows client can also resolve the issue. To do this, you can use the “net stop smb” and “net start smb” commands in the Command Prompt.
If the issue persists, it may be helpful to examine the Samba logs on the server to determine the root cause of the problem.
To disable Opportunistic Locking in Samba, you need to modify the smb.conf file on the Samba server. The smb.conf file is the main configuration file for Samba, and it’s usually located in the /etc/samba directory on Linux systems.
Here are the steps to disable Opportunistic Locking in Samba:
- Open the smb.conf file: You can open the smb.conf file using a text editor, such as nano or vim, by running the following command in the terminal:
sudo nano /etc/samba/smb.conf
- Locate the “oplocks” parameter: Look for the “oplocks” parameter in the smb.conf file. If the parameter does not exist, you can add it to the end of the file.
- Set the “oplocks” parameter to “False”: Once you have located the “oplocks” parameter, set its value to “False” by adding the following line:
oplocks = False
- Save and close the smb.conf file: After making the changes, save the file and close the text editor.
- Restart Samba: To apply the changes, you need to restart the Samba service by running the following command in the terminal:
sudo systemctl restart smb
Once you have completed these steps, Opportunistic Locking should be disabled in Samba, and Windows clients should no longer cache file data on their local machines, which should help resolve any issues with not refreshing files.
The “oplocks = False” configuration can be placed either under the [global] section or under a specific [share] section in the smb.conf file.
If you place the “oplocks = False” configuration under the [global] section, it will disable Opportunistic Locking for all shares on the Samba server.
[global]
oplocks = False
If you place the “oplocks = False” configuration under a specific [share] section, it will disable Opportunistic Locking only for the share specified in that section. For example:
[share1]
oplocks = False
[share2]
...
In this case, Opportunistic Locking will be disabled only for the share1 share and not for share2.
It’s important to note that if you place the “oplocks = False” configuration under a specific [share] section, and if that share does not have the “oplocks” option set, Samba will use the global setting from the [global] section.
Và mình đã sửa Dockerfile như sau:
https://github.com/mrnim94/samba/commit/b57c89000b351d6de9370a46a4bc2ba68d292437
Recheck configuration of SMB/Samba server.
The testparm command is a utility that verifies the syntax and consistency of the Samba configuration file (smb.conf)
To check the maximum number of connections to a Samba server, you can use the testparm command
Here’s how to check the maximum number of connections to a Samba server using the testparm command:
- Open a terminal or command prompt on the computer that is running the Samba server.
- Run the following command:
testparm -v | grep "max connections"This command will display the maximum number of connections allowed by the Samba server.
If the max connections parameter is not specified in the smb.conf file, the default value is set to 0, which means an unlimited number of connections are allowed.
Current connections to the Samba server
Alternatively, you can also check the smbstatus command to see the current connections to the Samba server:
- Open a terminal or command prompt on the computer that is running the Samba server.
- Run the following command:
smbstatus -bThis command will display the current connections to the Samba server, including the username, IP address, and other information about each connection.
Locked files: Pid User(ID) DenyMode Access R/W Oplock SharePath Name Time -------------------------------------------------------------------------------------------------- 66 100 DENY_WRITE 0x120089 RDONLY NONE /smbshare pvc-321a62b5-8ea0-4f60-925f-b688155926de/93bbbd0b73d784ad5762ca7a50539016118665d6b97fdff7db95634f1e455a25 Sun Mar
The columns in the table represent the following information about locked files in a Samba server:
- Pid: Process ID of the process that has locked the file.
- User(ID): User and user ID of the process that has locked the file.
- DenyMode: The type of lock placed on the file, which can be “DENY_NONE” (no locks), “DENY_READ” (file is locked for reading), “DENY_WRITE” (file is locked for writing), or “DENY_ALL” (file is locked for both reading and writing).
- Access: The type of access requested on the file, which can be “READ” or “WRITE”.
- R/W: Indicates whether the lock is for reading (R) or writing (W).
- Oplock: Opportunistic locking status of the file, which can be “NONE” (opportunistic locking is disabled), “BATCH” (opportunistic locking is enabled for batch operations), or “EXCLUSIVE” (opportunistic locking is enabled exclusively).
- SharePath: The path of the shared folder where the file is located.
- Name: The name of the locked file.
- Time: The time when the lock was placed on the file.
Unlock the files on Samba server.
1) Kill user’s PID
Better option is to kill Samba process which is locking the file. Samba assign a separate process to every connected user. This way only files locked by that particular user will be unlocked. It won’t drop the connection in the same way as restarting samba does.First run:
smbstatus
and find the locked file. Note PID number (first column)
If list is very long and you are struggling to find the file run:
smbstatus | grep -i filename.ext
where filename.ext is name of the locked file.
This will filter the smbstatus results and return only lines with your locked file.
Now run smbstatus again (or return to the first query) and in the top table find the user with the PID number you noted. This will be the user the who is locking the file.
You can also check what other files this user has opened by inspecting smbstatus results or running:
smbstatus | grep 14899
where 14899 user’s PID number
If possible, ask user to close all files opened from this particular server.
Finally, to unlock all files this user has opened, run
kill 14899
where 14899 user’s PID number
2) Restart Samba
Quickest and most dirty way is to restart samba. Problem is that all users will be temporarily disconnected from the server, potentially resulting in data loss. To restart samba, run:
/etc/init.d/samba restart
Command may be slightly different depending on your Linux distribution.
3) other ways
The below way can’t work
By command
Khi bản sử dụng câu lệnh smbstatus
bạn sẽ thấy các file đang bị block bởi client
Locked files: Pid User(ID) DenyMode Access R/W Oplock SharePath Name Time -------------------------------------------------------------------------------------------------- 66 100 DENY_WRITE 0x120089 RDONLY NONE /smbshare pvc-321a62b5-8ea0-4f60-925f-b688155926de/93bbbd0b73d784ad5762ca7a50539016118665d6b97fdff7db95634f1e455a25 Sun Mar 5 05:27:05 2023 66 100 DENY_WRITE 0x120089 RDONLY NONE /smbshare pvc-321a62b5-8ea0-4f60-925f-b688155926de/93bbbd0b73d784ad5762ca7a50539016118665d6b97fdff7db95634f1e455a25 Sun Mar 5 05:27:05 2023 66 100 DENY_WRITE 0x120089 RDONLY NONE /smbshare pvc-321a62b5-8ea0-4f60-925f-b688155926de/93bbbd0b73d784ad5762ca7a50539016118665d6b97fdff7db95634f1e455a25 Sun Mar 5 05:27:05 2023 66 100 DENY_WRITE 0x120089 RDONLY NONE /smbshare pvc-321a62b5-8ea0-4f60-925f-b688155926de/f5aa4fd368028e1f5333dfdbfa7b6b0e1c199bbb93858a596e508898a5298d25 Sun Mar 5 07:15:32 2023
Bạn có thể unblock bằng câu lệnh bên dưới.
chattr -i /smbshare/pvc-321a62b5-8ea0-4f60-925f-b688155926de/f5aa4fd368028e1f5333dfdbfa7b6b0e1c199bbb93858a596e508898a5298d25By config (have not check yet)
blocking locks = yes
In Samba, the blocking locks option controls whether or not the server should use blocking locks when a client requests a lock on a file.
When blocking locks is set to yes, the server will block other clients from accessing the locked file until the client that has the lock releases it. This means that other clients may experience delays when trying to access the locked file.
When blocking locks is set to no, the server will not block other clients from accessing the locked file. Instead, it will immediately return an error message to any clients that attempt to access the locked file.
By default, blocking locks is set to yes in Samba. Changing this option to no may improve performance in some situations, but it may also increase the risk of file corruption or data loss if multiple clients try to modify the same file simultaneously.
It’s important to consider the specific needs and requirements of your network when deciding whether or not to enable blocking locks on your Samba server.
Disable encryption of SMB traffic on a Samba server
the configuration file you would need to modify depends on the specific implementation of the SMB server.
For example, if you are using the Samba server, the configuration file is typically located at /etc/samba/smb.conf. You can modify this file using a text editor to adjust the SMB server configuration settings.
To disable encryption of SMB traffic on a Samba server, you can add or modify the following lines in the [global] section of the smb.conf file:
server signing = auto server min protocol = SMB2 smb encrypt = off
server signing: This setting controls whether the server should sign SMB packets using a digital signature. Packet signing helps to ensure that the packets have not been modified or tampered with in transit. The value “auto” means that the server will negotiate signing with the client, meaning that if the client supports packet signing, the server will use it, otherwise it will not.server min protocol: This setting specifies the minimum SMB protocol version that the server will support. The value “SMB2” specifies that the server will only support SMB version 2 or higher. SMB2 includes several security enhancements compared to earlier SMB versions, such as improved packet signing and encryption.smb encrypt: This setting controls whether SMB traffic should be encrypted. The value “off” means that encryption will be disabled, allowing unencrypted communication between the server and clients.
Specify the particular version on the Samba server.
Đầu tiên bạn cần vào trang này để tìm version samba phù hợp,
https://pkgs.org/download/samba
https://pkgs.alpinelinux.org/packages?name=samba&branch=v3.18&repo=&arch=&maintainer=
apk add samba=4.16.10-r0Setting up SMB Multi-Channel
Mình sẽ lưu configurations.
https://codeinsecurity.wordpress.com/2020/05/18/setting-up-smb-multi-channel-between-freenas-or-any-bsd-linux-and-windows-for-20gbps-transfers/
server multi channel support = yes aio max threads = 500 aio read size = 1 aio write size = 1
Another Samba on docker
https://github.com/crazy-max/docker-samba
https://github.com/andyzhangx/samba/tree/windows-fix
Issues
Error response from daemon: error while creating mount source path … mount: file exists
https://github.com/kubernetes-csi/csi-driver-smb/issues/633
MountVolume.MountDevice failed for volume "pvc-fa6a6a11-7a56-4309-b712-72dd1a4a4a8a" : rpc error: code = Internal desc = volume(172.20.206.119/share#pvc-fa6a6a11-7a56-4309-b712-72dd1a4a4a8a#) mount "//172.20.206.119/share/pvc-fa6a6a11-7a56-4309-b712-72dd1a4a4a8a" on "\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fa6a6a11-7a56-4309-b712-72dd1a4a4a8a\\globalmount" failed with NewSmbGlobalMapping(\\172.20.206.119\share\pvc-fa6a6a11-7a56-4309-b712-72dd1a4a4a8a, c:\var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fa6a6a11-7a56-4309-b712-72dd1a4a4a8a\globalmount) failed with error: rpc error: code = Unknown desc = NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : Network access is denied. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n + CategoryInfo : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \r\n pping], CimException\r\n + FullyQualifiedErrorId : Windows System Error 65,New-SmbGlobalMapping\r\n \r\n", err: exit status 1Mình gặp trường hợp là node linux và pod linux work bình thường chỉ có windows là bị.
nó là do dòng này.
smb encrypt = off' >>$file
